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Abstract 

B etween 2007 and 2009, reports were released on the results of three sepa- 
rate large-scale random assignment studies of the effectiveness of school- 
based mentoring programs for youth. The studies evaluated programs 
implemented by Big Brothers Big Sisters of America (BBBSA) affiliates (Her- 
rera et al., 2007), Communities In Schools of San Antonio, Texas (Karcher, 
2008), and grantees of the U.S. Department of Education’s Student Mentoring Program 
(Bernstein et al., 2009). Differences in the findings and conclusions of the studies have 
led to varying responses by those in practice and policy roles. The results of the BBBSA 
trial led the organization to undertake an initiative to pilot and evaluate an enhanced 
school-based mentoring model. Findings of the Student Mentoring Program evalua- 
tion were cited as a reason for eliminating support for the program in the FY 2010 
federal budget (Office of Management and Budget, 2009). In this report, we present 
a comparative analysis of the three studies. We identify important differences across 
the studies in several areas, including agency inclusion criteria, program models, 
implementation fidelity and support, and criteria utilized in tests of statistical signifi- 
cance. When aggregating results across the studies using meta-analytic techniques, 
we find evidence that school-based mentoring can be modestly effective for improving 
selected outcomes (i.e., support from non-familial adults, peer support, perceptions 
of scholastic efficacy, school-related misconduct, absenteeism, and truancy). Program 
effects are not apparent, however, for academic achievement or other outcomes. 

Our analysis underscores that evidence-based decision-making as applied to youth 
interventions should take into account multiple programmatic and methodological 
influences on findings and endeavor to take stock of results from the full landscape of 
available studies. 
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From the Editors 

Is the program effective? Should we continue to support it? Federal, state, and local 
government officials regularly ask these questions regarding programs of all shapes 
and sizes. More often than not, the research does not provide a definitive answer. This 
is the situation in which researchers and policymakers find themselves when consider- 
ing school-based mentoring programs. Since 2007, findings from three large random 
assignment evaluations of school-based mentoring programs have been released-with 
differing findings across the studies. 

In this issue of Social Policy Report, Wheeler, Keller, and DuBois critically 
examine and synthesize findings across these three large studies. They discuss both 
programmatic and methodological issues that may account for the different findings. 
They also use meta-analytic techniques to combine findings across studies as a strat- 
egy for better understanding the general impact of school-based mentoring. Their 
critique will inform policymakers and researchers about the effectiveness of school- 
based mentoring programs. 

There are no easy answers, though. As Wheeler, Keller, and DuBois write, “a 
simple answer as to whether school-based mentoring programs ‘work’ will inevitably 
remain elusive.” Much depends on the program characteristics, outcomes of interest, 
and evaluation design. The authors note that, “Given the diversity of school-based 
mentoring models and programs and the fact that mentoring is an individualized 
intervention, planners must consider which model will work for which students under 
which circumstances.” 

To better understand the implications of the comparative analysis and discus- 
sion of findings from these three studies, we offer commentaries from three different 
perspectives. The first commentary by Roth places school-based mentoring in the 
broader context of how children spend their time out of school and notes some com- 
mon themes. The second commentary by Foster addresses the important policy ques- 
tion of cost-effectiveness. The last commentary by Meyer provides an on-the-ground 
perspective from a public school district administrator responsible for a school-based 
mentoring program (that is not one of the three program types reviewed). 

Finally, this SPR raises the issue of bias (real or perceived) sometimes inherent 
in research. As evident in the author biographical statements, Wheeler, Keller, and 
DuBois have conducted research related to youth mentoring and, in particular, have 
worked with one of the three school-based mentoring programs included in their 
comparative analysis. Knowing this, one might question their objectivity in conducting 
this work. It is, after all, very important for research to be as unbiased as possible. 

Yet, researchers with both knowledge and practical experience may be the very ones 
who should share their expertise with program administrators and policymakers. 
Sometimes the research-to-practice link complicates the lines of objectivity. Instead 
of considering this a “fatal flaw”, as editors we chose to carefully review the paper 
for any perceived bias in favor of or against a particular program and, as always, 
sought reviews of experts in the field. The authors, up-front about the issue from the 
beginning, were very responsive to our editorial suggestions. Yes, we always need to 
be wary of bias-and we also need researchers with program expertise. In closing, we 
hope that readers will find this paper and the commentaries useful in further under- 
standing the effectiveness of school-based mentoring programs and answering the 
seemingly simple yet often elusive question of program effectiveness. 

-Kelly L. Maxwell (Issue Editor) 

Samuel L. Odom (Lead editor) 
Donna Bryant, Anne Hainsworth 
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Review of Three Recent 

Randomized Trials of School-Based Mentoring 

Making Sense of Mixed Findings 



D uring the last fifteen years, mentoring has 
become one of the country’s most popu- 
lar interventions to improve the lives of 
disadvantaged and at-risk youth (Walker, 
2007). Although mentoring programs share 
an emphasis on cultivating supportive rela- 
tionships between young persons and non-parental adults 
or older peers (DuBois & Karcher, 2005), they vary widely 
in their goals, populations served, and delivery formats 
(Karcher, Kuperminc, Portwood, Sipe, & Taylor, 2006; 
Keller, 2007). One of the newest and fastest growing pro- 
gram models is school-based mentoring (SBM). Of more 
than 4,700 programs in a national database of mentoring 
programs, approximately one in four 
(28%) use a school-based format (K. 

Zappie-Ferradino, personal commu- 
nication, January 6, 2010). A central 
distinguishing feature of school-based 
mentoring programs is that meetings 
between youth and their mentors 
typically are structured to take place 
only in the school setting. Accordingly, 
program goals routinely include im- 
provements in academic performance 
and school-related behavior (Portwood 
& Ayers, 2005). 

Recently, three relatively large- 
scale randomized controlled trials 
(RCTs) of the effectiveness of school- 
based mentoring programs have been 
conducted (Bernstein, Dun Rappaport, 

Olsho, Hunt, & Levin, 2009; Herrera, 

Grossman, Kauh, Feldman, & McMak- 
en, 2007; Karcher, 2008). The primary 
reports of these studies reached differing conclusions 
regarding the effectiveness of school-based mentoring as 
assessed at the end of one school year of participation. 

An evaluation of the Big Brothers Big Sisters of America 
(BBBSA) school-based mentoring program concluded that, 



“Impacts measured after one school year of involvement 
in the BBBS SBM program showed that ‘Littles’ (youth 
assigned to receive mentors) improved in a range of 
school-related areas, including their academic attitudes, 
performance and behaviors” (Herrera et al. , 2007, p. 67). 
An evaluation of school-based mentoring provided within 
Communities In Schools of San Antonio’s (CIS-SA) program 
concluded that “school-based mentoring as typically 
implemented within a multi-component program may be 
of limited value for students in general and most helpful 
to elementary school boys and high school girls” (Karcher, 
2008, p.112). Finally, an evaluation of programs funded 
through the U.S. Department of Education’s Student 

Mentoring Program found that the pro- 
grams studied “did not lead to statisti- 
cally significant impacts on students 
in any of the three outcome domains 
[prosocial behavior, problem behavior, 
academic achievement]” (Bernstein et 
al., 2009, p. xx). 

Organizational and policy re- 
sponses have varied depending on 
which report is being used as a primary 
point of reference. Drawing on the 
evaluation of its school-based mentor- 
ing program, BBBSA has continued to 
support implementation of the program 
by its affiliates and is investing signifi- 
cant resources in strengthening the 
program (BBBSA, 2008). By contrast, 
the administration of President Barack 
Obama cited findings of the Depart- 
ment of Education evaluation when 
it proposed eliminating funding for 
the Student Mentoring Program from the federal budget 
(Boyle, 2009; Office of Management and Budget, 2009). 

Our goal in this report is to critically examine and 
synthesize findings from the three studies, thereby offer- 
ing a stronger foundation for future decision-making re- 
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garding school-based mentoring as an intervention strate- 
gy. After providing some further background, we undertake 
a comparative examination of the three recent evaluations 
with respect to features of both the programs investigated 
and the research methodologies used. We draw attention 
in particular to similarities and differences that could 
be important in accounting for variation in findings and 
conclusions across the reports. We then synthesize findings 
from the studies using meta-analytic techniques to clarify 
overall trends in the impact of school-based mentoring 
programs on youth outcomes. We conclude by considering 
the implications of our analysis for current understanding 
of school-based mentoring as an intervention strategy for 
at-risk youth. In doing so, we highlight several issues that 
are relevant more broadly to the ongoing discussion and 
debate regarding optimal approaches to evidence-based 
decision-making in policy and practice. 

Background 

Reviews of research on factors that foster resilience 
among vulnerable or at-risk youth consistently identify a 
close connection with a non-parental adult as a protec- 
tive factor (Garmezy, Masten, & Tellegen, 1984; Masten & 
Coastworth, 1998; Werner, 1995). Theoretically, mentor- 
ing relationships may foster resilience by counteracting 
or offsetting the negative consequences of risk factors 
to which youth are exposed (compensatory effect), by 
ameliorating the association between risk exposure and 
outcomes (risk-protective effect), and by enhancing the 
effects of other protective factors in the youth’s life 
(protective-protective effect; M. A. Zimmerman, Bingen- 
heimer, & Behrendt, 2005). More specifically, a youth’s 
relationship with a non-parental adult or older peer who 
serves as a mentor may enhance coping and promote 
positive socio-emotional, cognitive, and identity develop- 
ment (Rhodes, 2005; Rhodes, Spencer, Keller, Liang, & 
Noam, 2006). Social and emotional adjustment may be 
strengthened by providing youth with a secure attach- 
ment experience, enhancing interpersonal skills and emo- 
tion regulation, and offering opportunities for fun and 
diversion from stress. Cognitive skills may be improved 
through exposing the youth to new opportunities for 
learning, providing intellectual challenge and guidance, 
and encouraging academic achievement. Identity devel- 
opment may be fostered through role modeling, personal 
appraisals and feedback, and promoting participation in 
activities and settings that build the youth’s social and 
cultural capital. 



Because mentoring programs are most commonly 
structured as an individualized intervention based on a 
one-to-one relationship, their effects are likely to vary 
according to the quality of the relationships that are 
established (Keller, 2007; Rhodes et al. , 2006). In general, 
mentoring relationships that are of longer duration, have 
more frequent and consistent meetings, and are charac- 
terized by a strong emotional bond are associated with 
better youth outcomes (Rhodes & DuBois, 2006), although 
positive outcomes have also been associated with certain 
mentoring interventions that focus on relatively short- 
term relationships with presumably limited opportunity 
for emotional connection (Hughes, Cavell, Meehan, 

Zhang, & Collie, 2005). Evidence also suggests that men- 
toring relationships can have detrimental consequences 
when they end prematurely or are characterized by 
negative interactions or role modeling (Rhodes & DuBois, 
2006). 

Historically, youth mentoring programs have 
sought to create a personal relationship between an 
adult volunteer and a young person through encouraging 
mentoring pairs (commonly referred to as “matches”) 
to spend time together in a wide range of activities in 
the community (Baker & Maguire, 2005). Public/Private 
Ventures (P/PV) evaluated the effectiveness of the BBBSA 
community-based mentoring program over a decade ago 
in a landmark study in which 959 youth referred to eight 
BBBSA agencies were followed over an 18- month period 
(Tierney, Grossman, & Resch, 1995). Compared to those 
who were assigned randomly to a wait list control group, 
youth assigned to receive a mentor were less likely to 
report onset of drug and alcohol use, skipping school, or 
hitting someone and were more likely to report improved 
parental relationships. These results generated great 
enthusiasm for mentoring on the part of politicians, poli- 
cymakers, media, and the public at large and provided 
a catalyst for not only growth in the number of youth 
served through BBBSA but also for the proliferation of 
new mentoring programs in local communities (Rhodes & 
DuBois, 2006). The findings of this evaluation have come 
to be viewed more cautiously as subsequent analyses 
have highlighted the modest size of the effects reported 
(Rhodes & DuBois, 2006). Concerns also have been raised 
regarding their indiscriminate use to make claims of 
effectiveness for a wide range of relationship-based 
interventions for youth (Boyle, 2007). 



Social Policy Report V24 #3 



4 



School -Based Mentoring 



Emergence of School-Based Mentoring 

Coinciding with these developments, many youth 
mentoring organizations began partnering with school 
districts across the United States to provide mentoring 
to youth in schools (Herrera et al., 2007). School-based 
programs grew at a time when there was increasing con- 
sensus that schools should be centers for a wide range 
of social, psychological and health services (Dryfoos, 
1991). Furthermore, the No Child Left Behind (NCLB) Act 
of 2001 began to place increased pressure on schools to 
produce improved academic outcomes as demonstrated 
through standardized test scores, dampening the enthu- 
siasm of schools for investing in programs not perceived 
to be aligned with this goal (Portwood & Ayers, 2005). 
Thus, the rise of school-based mentoring has been some- 
what contingent on its perceived promise to improve 
academic outcomes. 

Recent growth in school-based mentoring has been 
supported in part by initiatives funded through the U.S. 
Department of Education. The Student Mentoring Pro- 
gram, first authorized in the NCLB Act of 2001, grew from 
a $17 million per year appropriation to approximately $50 
million by 2004 (U.S. General Accounting Office, 2004). 
Illustrating this growth, between 1999 and 2006 the 
number of youth served through school-based mentoring 
in BBBSA affiliates increased from 27,000 to 126,000 (Her- 
rera et al., 2007). A recent survey of a nationally repre- 
sentative sample of adults generated an estimate that 
over one-quarter (29%) of the approximately three million 
adults who volunteered in mentoring programs did so in 
school-based settings (MENTOR, 2005). 

In the typical school-based mentoring program, 
staff members match volunteer mentors with students 
from the host school on a one-to-one basis. These 
matches then meet regularly (e.g., weekly) at the 
school throughout the academic year. Mentors and stu- 
dents may spend their time together doing homework, 
talking, or participating in games, arts, crafts, and 
other activities (Portwood & Ayers, 2005). Unlike tutors, 
school-based mentors participate in a range of activities 
with their student mentees, and the provision of aca- 
demic assistance is only one aspect of the relationship 
(Herrera et al., 2007). 

A school-based model has been attractive to 
mentoring agencies for several reasons (Herrera, 2004; 
Herrera et al., 2007; Portwood & Ayers, 2005). Such 
programs, for example, may attract a broader pool of 
volunteers by offering greater structure and/or a more 
limited and predictable time commitment than is typical 



of community-based mentoring programs. Research sug- 
gests that school-based mentors are in fact more varied 
in age and racial and ethnic heritage than those who 
volunteer for community-based programs (Herrera, 2004; 
Herrera et al., 2007). School-based mentoring similarly 
may offer greater ability to reach particular populations 
of youth due to reduced requirements for parental in- 
volvement (Herrera et al., 2007). School-based programs 
also have been thought to reduce staff investment in 
mentor screening and supervision, thereby serving more 
children with lower costs (Portwood & Ayers, 2005). 

In particular, because meetings between students and 
mentors in the school environment are more likely to be 
observed by school personnel and mentoring program 
staff, there may be fewer safety risks to address in 
mentor screening as well as more efficient monitoring of 
relationships once established. A detailed examination of 
the actual costs of school-based mentoring within BBBSA, 
however, found that annual costs per youth served ($987) 
did not differ appreciably from those associated with the 
organization’s community-based program ($1,088; Her- 
rera et al., 2007). 



Prior Evaluations of School-Based Mentoring Programs 

A 2002 meta-analysis of youth mentoring program evalu- 
ations found a trend toward smaller effects on youth 
outcomes for programs in which interactions between 
mentors and youth were limited to the school setting 
(DuBois, Holloway, Valentine, & Cooper, 2002). At the 
time of that review, however, school-based mentoring 
was a very recent form of intervention, and programs 
arguably had not matured enough to demonstrate their 
potential. Subsequent evaluations have reported evi- 
dence of positive effects of school-based mentoring on 
various youth outcomes, including classroom behavior 
and connectedness to school, family, and the community 
(King, Vidourek, Davis, & McClellan, 2002; Lee & Cra- 
mond, 1999; Portwood, Ayers, Kinnison, Waris, & Wise, 
2005). The evaluations are limited methodologically, 
however, by relatively small sample sizes and lack of 
consistent utilization of experimental designs (Randolph 
& Johnson, 2008). 



Recent Randomized Controlled Trials 

These limitations have been addressed recently through 
the three recent random assignment evaluations of the 
BBBSA school-based mentoring program (Herrera et 
al., 2007), mentoring provided through Communities In 
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Schools of San Antonio, Texas (CIS-SA; Karcher, 2008), and 
programs funded through the U.S. Department of Educa- 
tion’s Student Mentoring Program (SMP; Bernstein et al., 
2009). As noted previously, the findings and conclusions 
of these studies as well as the policy and organizational 
responses to them have varied. The BBBSA study reported 
positive program effects on several outcome measures 
at the end of one school year. These included teacher 
reports of academic performance, quality of class work, 
number of assignments completed, engagement in seri- 
ous school-related misconduct, and truancy as well as 
youth reports of relationships with non-parental adults, 
starting to skip school, and perceived scholastic efficacy. 
These effects were generally no longer evident when 
youth originally enrolled in the study were reassessed in 
late fall of the following school year, approximately 15 
months after the baseline assessment, when nearly half 
of the youth in the treatment group (48%) were no longer 
receiving mentoring. With the release of the study, BBBSA 
recommended that its affiliate agencies adopt strategies 
aimed at enhancing the effectiveness of its school- based 
program. These included enhanced volunteer training 
and support given to matches (e.g., by using assessments 
of relationship quality to determine which matches need 
greater assistance), lengthening match relationships (e.g., 
by asking participants for a one calendar year commitment 
instead of one school year), and providing agency sup- 
port through intervening summer months (BBBSA, 2007). 
The organization’s recommendations were informed by 
findings from the Herrera et al. (2007) report suggesting 
that certain program practices were associated with more 
favorable youth outcomes. These recommendations sub- 
sequently have been incorporated into a revised school- 
based program model that is being piloted and evaluated 
in 23 BBBSA agencies (BBBSA, 2009). 

The CIS-SA study investigated the influence of men- 
toring over and above the standard set of individualized 
support services offered to students through the Com- 
munities In Schools model. It reported positive overall ef- 
fects of mentoring participation on outcomes pertaining 
to connectedness to peers, self-esteem, and perceived 
social support from friends, but not on measures of aca- 
demic achievement or social skills (Karcher, 2008). Based 
on results of the study, CIS-SA made several changes to 
its program model and practices (I. Garcia, personal 
communication, February 12, 2010). 

As indicated earlier, the evaluation of programs 
implemented by SMP grantees reported an absence of 
any overall program effects on youth outcomes, which 



were assessed in the domains of academic achievement 
and engagement, interpersonal relationships and per- 
sonal responsibility, and high-risk or delinquent behav- 
ior (Bernstein et al., 2009). In the wake of the Obama 
administration recommendation to eliminate funding for 
the program, funding already designated for the final 
year of the program was retained (Fiscal Year 2009), but 
no new grants were awarded (Fernandes, 2009). 

With these developments, school-based mentoring 
appears to be at a crossroads. Depending on how find- 
ings and conclusions from the three recent evaluations 
are interpreted or weighted, arguments seemingly could 
be made for or against continued investments in school- 
based mentoring as a strategy for promoting resilience 
among at-risk youth. In the remainder of this report, we 
provide an in-depth comparison of both the programs 
evaluated in the three trials and study methodologies, 
with special attention given to those differences that 
help to illuminate reasons for differences in findings 
across the reports. We then report our findings when us- 
ing meta-analysis to integrate findings across the investi- 
gations. In doing so, our aim is to strengthen the founda- 
tion for informed decision-making rather than to provide 
a definitive answer to questions about the effectiveness 
of school-based mentoring programs. 

Comparative Analysis of the 

BBBSA, CIS-SA, and SMP Randomized Trials 

of School-Based Mentoring 

The findings of any program evaluation can be expected to 
be fundamentally shaped both by the nature and intensity 
of the intervention being investigated and by the quality 
and consistency with which the intervention was imple- 
mented (Durlak & DuPre, 2008). Methodological consider- 
ations regarding the evaluation itself, ranging from study 
design to data analysis, have the potential to be similarly 
influential (Shadish, 2002). In the following sections, we 
consider each of these potential sources of influence on 
findings of the three school-based mentoring trials. 



Program Characteristics and Implementation 

The basic description of school-based mentoring that we 
have provided encompasses a wide range of potential 
program variations. With respect to fidelity of program 
implementation, it is also noteworthy that all the pro- 
grams investigated were in existence before the respec- 
tive studies were initiated and that the researchers 
exerted no significant control over program operations. 
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As such, each of the SBM studies is best categorized as 
an effectiveness trial (i.e., an evaluation of a program’s 
impact when implemented without significant external sup- 
port or involvement from researchers) as opposed to an ef- 
ficacy trial (i.e., an evaluation in which researchers seek to 
directly ensure a high fidelity of program implementation; 
Flay, 1986). Their findings thus may be especially useful for 
indicating what types of program effects realistically can 
be expected under typical circumstances of implementa- 
tion. Such estimates may be attenuated from those that 
would be possible under circumstances in which more 
intensive supports are provided for program delivery. 

Program models. Program characteristics and sum- 
mary indicators of service delivery are shown in Table 1 . 
The programs included in each evaluation differed in the 
extent to which they were based on a uniform or more 
broadly sponsored model of school-based mentoring. The 
BBBSA study focused on the organization’s school-based 
mentoring model as implemented by several of its af- 
filiate agencies. BBBSA affiliate agencies are required to 



adhere to a set of national practice standards. They also 
are encouraged to implement a “Service Delivery Model” 
that contains non-binding operational guidelines for 
each step required to establish and support school-based 
mentoring matches. The San Antonio chapter of CIS had 
considerable experience implementing school-based 
mentoring. The national CIS organization, however, lacks 
a formal school-based mentoring program model. In 
this study, findings were examined across programming 
being implemented by the San Antonio CIS agency in 19 
schools within the same large school district. Significant 
school-level variation in outcome effects was observed 
in this trial (Karcher, 2008), highlighting the potential 
for differences in program delivery to be influential. 

The SMP was a federal initiative for competitive funding 
of school-based mentoring programs offered by a wide 
range of local organizations, including community-based 
non-profit organizations and school districts. Programs 
were expected only to follow the general requirements 
of the authorizing legislation, which mandated several 



Table 1 

Program Characteristics and Implementation in the Three Studies 





BBBSA 


CIS-SA 


SMP 


PROGRAM CHARACTERISTICS 


Mentor training 


None required in national program 
standards/service delivery model 3 - b 


1 hour mandatory, 
additional training optional 


Legislation requires 
mentor training 


Length and frequency 
of match meetings 


No requirements a - b 


1 hour weekly 


No requirements 


Minimum relationship 
duration commitment 


1 school year c 


1 school year 


Unspecified in legislation 


Match support 


Monthly for mentors; bi-monthly 
for mentees a 


CIS-SA case managers were 
available at the schools to help 
mentors, but there were no 
specific support requirements d 


General requirement in 
legislation 


PROGRAM IMPLEMENTATION 


Programmatic experience in 
school-based mentoring 


9.5 years (median = 7.5 years) d - e 


14 years 


6.1 years (median = 5.0 years) e 


Average amount of pre-match 
mentor training 


-45 minutes f 


50 minutes d 


3.4 hours 8 


Average number of months of 
mentoring received 


5.3 months 


5.9 months d 


5.8 months 


Average total amount of 
mentor-mentee interaction 


-17 hours f -s 


-10 hours d - h 


-23 hours 8 



Note. All data presented have been gathered from the three RCT reports (Bernstein et al., 2009; Herrera et al., 2007; Karcher, 2008), unless otherwise noted. 
Tilda is used in the table to denote approximations. a (BBBSA, 2003). b (BBBSA, 2004). c Based on personal communication with national office staff. 
d Based on personal communication with principal investigator. e Based on reported characteristics of the specific programs included in the randomized trial. 
f Figures estimated through secondary calculations of data presented in Herrera et al., 2007. 8 Based on reports from mentors completing year-end survey. 
h Based on logs completed by mentors after match meetings. 
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program practices such as volunteer training and match 
supervision, but stipulated few operational details, such 
as the required frequency of contacting program partici- 
pants. It may, in fact, be more appropriate to view the 
SMP study as a test of a funding initiative with certain 
parameters for allowable programming rather than as an 
evaluation of a particular mentoring program. A clearly 
delineated model established within a broader organiza- 
tional framework may help to ensure consistent adoption 
of a desirable and coherent set of practices by programs. 
Accordingly, the preceding differences seem most likely 
to have disadvantaged the CIS-SA and SMP trials relative 
to the BBBSA trial in terms of their potential for reveal- 
ing robust effects of school-based mentoring. 

Implementation. Differences are also apparent 
with respect to program implementation (see Table 1). 
Some degree of implementation support was available to 
the programs included in the BBBSA trial from BBBSA na- 
tional office staff, in the CIS-SA trial from the principal 
investigator, and in the SMP trial from the National Men- 
toring Center via a contract from the U.S. Department 
of Education. However, only in the BBBSA trial was there 
additional implementation support directed specifically 
toward those programs included in the trial. It was also 
the only trial in which an effort was made to select 
participating agencies that would be equipped to imple- 
ment school-based mentoring programs with a higher 
level of fidelity. Study inclusion criteria included at least 
four years of experience delivering school-based mentor- 
ing, strong agency leadership, and strong, established 
relationships with participating schools (Herrera et al. , 

2007) . In contrast, the primary investigator of the SMP 
study, drawing on his experience with this trial, charac- 
terized school-based mentoring as a “cottage industry... 

A lot of these [mentoring programs] are sprouting out of 
the ground, and then they disappear. I mean, we went to 
send the results back to everybody, and we couldn’t find 
two of our grantees. They didn’t exist anymore” (Boyle, 
2009). In view of the well-established importance of im- 
plementation for program effectiveness (Durlak & DuPre, 

2008) , these differences likely increased the potential of 
the BBBSA trial to reveal evidence of positive effects of 
BBBSA’s school- based mentoring program. 

Mentors in the SMP study reported the greatest 
amount of pre-match training (3.4 hours). In view of 
evidence linking such training to higher quality mentor- 
ing relationships (Herrera, Sipe, McClanahan, Arbeton, & 
Pepper, 2000), this difference has potentially important 
implications. The average number of months of mentor- 



ing that youth received was comparable across the three 
trials (ranging from 5.3 to 5.9 months). Notably, however, 
average amounts of mentor-mentee contact in the BBBSA 
and SMP studies (17 and 23 hours, respectively) were 
substantially more than those reported for the CIS-SA study 
(10 hours). This variation may be somewhat overstated due 
to different reporting sources and calculations across stud- 
ies. It is nonetheless noteworthy given that more frequent 
contact tends to be associated with greater feelings of 
closeness in mentoring relationships (Parra, DuBois, Nev- 
ille, Pugh-Lilly, & Povinelli, 2002) as well as more favor- 
able youth outcomes (DuBois et al., 2002). 



Methodological Comparisons Among the Studies 
Sample characteristics. The characteristics of the agen- 
cies, school, youth, and mentors that constituted the 
sample for each study are summarized in Table 2. As 
already noted, procedures for agency selection differed 
across studies in ways that have potentially important 
implications for observed program effects. It is also 
noteworthy that none of the investigations made use of 
a random selection process. Consequently, their findings 
may not generalize to the larger sets of agencies and 
schools in which SBM was being implemented within each 
program (e.g., SMP). Extrapolations to the overall field 
of school-based mentoring from the findings of any of the 
studies would be even more tenuous. 

Whereas the BBBSA and SMP samples included stu- 
dents from a range of racial and ethnic backgrounds, the 
CIS-SA sample was almost wholly made up of Hispanic/ 
Latino youth. The study sample is also distinguished as the 
only one to have a large percentage of mentees attending 
high school. Potential variations in program benefits along 
dimensions of either student ethnicity or age thus are ger- 
mane to consider in comparative analysis of findings across 
studies. The sample in each of the trials, furthermore, was 
limited to youth in grades 4 and higher despite the reality 
that younger students are served by many school-based 
mentoring programs as well as the evidence that interven- 
tions initiated earlier in childhood have greater impacts 
(Heckman & Masterov, 2007). A recent randomized trial of 
the Experience Corps program, which brings older adults 
into schools to tutor and mentor younger children who are 
at risk of academic failure, found that the program had 
significant positive effects on the reading skills of first- 
through third-grade students (Morrow-Howell, Jonson- 
Reid, McCrary, Lee, & Spitznagel, 2009). 

Finally, as shown in Table 2, participating youth in 
the three studies exhibited similar indicators of envi- 
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Table 2 

Sample Characteristics of the Three Studies 




BBBSA 


CIS-SA 


SMP 


AGENCIES AND SCHOOLS 


Number of agencies in each study 


10 BBBSA affiliates 


1 agency - CIS-SA 


32 (2/3 non-profits, 
1/3 school districts) 


Number of schools in each study 


71 public schools in rural and urban 
school districts across the US: 

41 elementary, 27 middle, and 3 high 
schools 


19 public schools in 1 large metro- 
politan school district: 7 elemen- 
tary, 5 middle, 7 high schools 


103 elementary and middle 
schools across the US 


YOUTH 


Overall sample size 


1,139 


525 


2,360 


Gender (% female) 


54% 


67% 


57% 


Race/ethnicity 








White 


37% 


2% 


23% 


Hispanic/Latino 


23% 


78% 


29% 


Black/ African-American 


18% 


9% 


41% 


Multi-race/other 


23% 


11% 


7% 


Grade 








4-5 


61% 


19% a 

(5 th grade only) 


42% 


6-8 


34% 


37% a 


44% 


9-12 


6% 

(9 th grade only) 


44% a 


14% 


Poverty status 


69% received free or reduced lunch 


Average family income less than 
$20,000 


86% eligible for free or 
reduced lunch 


Academic risk 


56% with difficulties in overall academic 
achievement or unable to do school- 
work at grade level 


100% met Texas criteria for at-risk 
of dropping out of school (denotes 
unsatisfactory test performance at 
younger ages) a ' b 


60% below proficiency in 
either reading/English lan- 
guage arts or math (or both) 


Delinquency risk 


21% of youth at baseline reported 
stealing something or substance use in 
the last three months 


— 


25% of youth at baseline re- 
ported delinquent behaviors 


MENTORS 


Gender (% female) 


72% 


73% 


72% 


Race/ethnicity 








White 


77% 


35% 


66% 


Hispanic/Latino 


6% 


54% 


10% c 


Black/African 


8% 


5% 


29% 


American Multi-race/other 


10% 


6% 


9% 


Student/Age status 








High school student 


48% 


0% 


18% 


College student 


18% 


70% 


23% 


Adult (under 65) 


33% 


28% 


56% 


Other 


1% 


2% 


3% 


Note. All data presented have been gathered from the three RCT reports (Bernstein et at., 2009; Herrera et at., 2007; Karcher, 2008), unless otherwise noted. 
Dash is used in the table to denote data not reported or able to be obtained. a Based on personal communication with principal investigator. b Texas Education 
Agency (2010). c Ethnicity surveyed separately from race-hence figures total more than 100%. 
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ronmental and individual risk status. Available evidence 
suggests greater effectiveness for mentoring programs 
directed toward youth with both environmental and in- 
dividual risk factors compared to those targeted toward 
youth exhibiting only individual risk (DuBois et al . , 2002). 
As such, findings of these studies may not generalize to 
programs that are directed predominantly to students 
on the basis of existing personal problems or difficul- 
ties. Also of note is that half of the mentors in the BBBSA 
study were high school students. In contrast, the ma- 
jority of mentors in the CSI-SA and SMP study samples 
were college students and adults under 65, respectively. 
As discussed below, the BBBSA study found evidence of 
weaker program impacts for youth matched with a high 
school student mentor. 

Study designs and analyses. Features of the study 
designs and measures for the three trials are summarized 
in Table 3. To help ensure unbiased estimates of the im- 
pact of assignment to participate in a school-based men- 
toring program, each investigation reports findings for 
an “intent-to-treat” model in which available outcomes 
for the entire treatment group are compared to those for 
the control group, regardless of whether each youth in 
the treatment group actually ended up being paired with 



a mentor or, if matched, the youth’s mentoring relation- 
ship continued to the end of the school year. This type of 
analysis is used to help avoid various misleading artifacts 
that can arise in intervention research. For example, if 
analyses in any of the trials had focused only on youth 
who were able to be matched and who received a full 
year of mentoring, results could have been misleading 
both because all youth intended to receive mentoring 
were not being considered and because youth in the 
treatment group could no longer be considered compa- 
rable to those in the control group. 

As shown in Table 3, in the SMP study a notably 
higher percentage of treatment group students did not 
receive mentoring, comparatively weakening its abil- 
ity to detect program effects. Of further note are the 
experiences of youth assigned to the control group in 
each study. In both the BBBSA and SMP studies, ap- 
proximately one-third of the youth in the control group 
reported receiving some form of mentoring from another 
source. This, too, would be expected to lessen the ability 
of these studies to identify effects of mentoring program 
participation. Qualifying this conclusion, however, is 
the reality that exposure to mentoring through alterna- 
tive sources may be commonplace and thus arguably an 



Table 3 

Study Designs and Measures of the Three Studies 





BBBSA 


CIS-SA 


SMP 


STUDY DESIGN 


Nature of 
control group 


No treatment; 34% reported meet- 
ing with “an adult or older student 
mentor, ‘buddy’ or ‘big’” in the 
previous six months; 3 1 child also 
inadvertently received a mentor 


Received supportive services 
through CIS-SA (i.e. , educational 
enhancement activities, supportive 
guidance, enrichment activities, 
and/or tutoring); 2% also inadver- 
tently received a mentor 


No treatment; 35% reported being 
mentored in a formal program 


Nature of 
treatment group 


Intended to receive mentoring; 
7% not matched with a mentor b 


Intended to receive mentoring plus 
supportive services listed above; 
10% not matched with a mentor c - d 


Intended to receive mentoring; 
17% not matched with a mentor c 


MEASURES 


Domains 


• Academic performance 

• Academic attitudes 

• Academic competency behaviors 

• School- related misbehavior 

• Social skills/relations 

• Antisocial behavior 

• Self-esteem 


• Grades 

• Connectedness 

• Self-esteem 

• Social skills 

• Social support 

• Hope and perceived mattering 


• Academic engagement and achievement 

• Interpersonal relationships 

• Personal responsibility, and 
community involvement 

• Juvenile delinquency/ 
participation in harmful activities 



Note. All data presented have been gathered from the three RCT reports (Bernstein et al., 2009; Herrera et al., 2007; Karcher, 2008). 
a Based on personal communication with principal investigator who also noted that youth may have misinterpreted this item due to lack of optimal wording. 
b At the time of the 1 5-month assessment, 5% of the treatment group was unmatched. c Includes youth who were matched but never met with their assigned mentors. 
d Additionally, approximately 22% of the mentors quit after only one or two meetings with their mentees. 
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appropriate benchmark against which to evaluate pro- 
gram effectiveness. Finally, in the CIS-SA study, all youth 
in the study (i.e., both control and treatment groups) 
were recipients of the same, non-mentoring supportive 
CIS services, making this a study of the effects of adding 
mentoring to a set of other services. 

The implications of this design feature 
are unclear given that the benefits of 
mentoring could either be largely re- 
dundant with those of other supports 
or combine with them synergistically 
to enhance impact (i.e., a protective- 
protective effect). 

Each study assessed outcomes 
in similar domains. There was, how- 
ever, noteworthy variation in specific 
measures and in the sources used to 
assess different outcomes. Whereas 
the BBBSA study relied on student 
and teacher reports, the CIS-SA and 
SMP studies used student reports 
and school records. Each source of 
data has potential strengths and 
weaknesses. In the BBBSA study, for 
instance, teacher reports of aca- 
demic performance may have been 
influenced by knowledge of which 
students were receiving mentoring. 

At the same time, teacher reports 
of proximal outcomes such as qual- 
ity of classwork completed may have 
been more sensitive to detecting 
relatively nuanced changes in performance than were 
the indices of achievement derived from school records 
(e.g., report card grades) used in the other two trials. 

The three studies also used differing criteria to 
evaluate the statistical significance of program ef- 
fects on outcome measures. The criteria employed in 
each trial reflect different thresholds for tolerating 
the probability of a Type I error (attributing an effect 
to the program that does not truly exist and is instead 
apparent due to chance characteristics of a particular 
sample). Minimizing risk for Type I error would be a 
priority when decision-makers (e.g., funding agencies) 
want to be sure to invest in truly effective programs. 
However, being more stringent with the probability of 
Type I error increases the chance of making a Type II 
error (failing to attribute an effect to a program that it 
actually does achieve). A decision-maker concerned with 



this latter possibility might set a more lenient threshold 
to avoid inadvertently failing to detect true program 
effects. The BBBSA study utilized the most lenient level 
of Type I probability for identifying effects as statisti- 
cally significant (p-value less than .10 with no correction 
for inflated error risk due to multiple 
tests of significance across outcomes), 
whereas the SMP study used the most 
stringent criterion (p-value of less 
than .05 after correction for multiple 
tests of significance within the same 
outcome domain). Metaphorically, if 
each study were a baseball umpire, 
the BBBSA study would have called 
strikes on the edge of home plate, 
CIS-SA would have called them just 
over the plate, and SMP would have 
called strikes only for balls pitched 
right down the middle. 

To better understand the con- 
sequences of these differences, we 
examined how findings of the studies 
would compare when uniformly apply- 
ing the criterion for statistical signifi- 
cance that was used in any one of the 
studies to all three trials. As shown in 
Table 4, this approach suggests notably 
greater consistency in program impacts 
across studies than is apparent from 
the original reports. For instance, us- 
ing the middle ground criterion em- 
ployed by the CIS-SA study, the BBBSA 
study would have reported significant impacts on seven 
outcomes, the SMP study five, and the CIS-SA study four. 
Likewise, had all trials employed the more conservative 
approach used in the SMP study, the BBBSA and CIS-SA 
studies would have reported significant effects on only 
two and one outcomes, respectively, thus closely parallel- 
ing the finding of no effects in the SMP trial. 

Each study also tested for differences in program 
impact across subgroups of participating youth. Relative 
to the large number of tests that were conducted, these 
analyses revealed limited and inconsistent evidence of 
subgroup differences in program impact. Illustratively, in 
the BBBSA and SMP trials girls appear to have benefited 
from program participation to a greater extent than boys 
on selected outcomes (Bernstein et al. , 2009; Herrera et 
al. , 2007). Yet, in the CIS-SA trial there is a more complex 
pattern of findings involving both gender and grade level 



... uniformly applying 
the same criterion 
for statistical 
significance ... 
suggests notably 
greater consistency 
in program 
impacts ... 



Social Policy Report V24 #3 



11 



School -Based Mentoring 




Table 4 

Study Findings as a Function of Different Criteria for Statistical Significance 



Statistical 

Significance 

Criterion 


BBBSA 


CIS-SA 


SMP 


O 

V* 

Q_ 


• Perceived scholastic efficacy 

• Non-parental adult relationships 

• Truancy (teacher report) 

• Truancy (youth report) 

• Overall academic performance 

• Written and oral language performance 

• Science performance 

• Quality of class work 

• Number of assignments completed 

• Engaging in serious school misconduct 


• Connectedness to peers 

• Global self-esteem 

• Self-in-the-present 

• Peer social support 


• Perceived scholastic efficacy 

• Non-parental adult 
relationships 

• Truancy 

• Absenteeism 

• Future orientation 


p<.05 


• Perceived scholastic efficacy 

• Non-parental adult relationships 

• Truancy (youth report) 

• Overall academic performance 

• Quality of class work 

• Number of assignments completed 

• Engaging in serious school misconduct 


• Connectedness to peers 

• Global self-esteem 

• Self-in-the-present 

• Peer social support 


• Perceived scholastic efficacy 

• Non-parental adult 
relationships 

• Truancy 

• Absenteeism 

• Future orientation 


p<.05 + Benjamini- 
Hochberg test 


• Quality of class work a 

• Number of assignments completed a 


• Self-in-the-present a 





Note. All data presented have been gathered from the three RCT reports (Bernstein et al., 2009; Herrera et al., 2007; Karcher, 2008), unless otherwise noted. 
a Estimated through secondary calculations of data presented in original documents. 



in which differential program effects instead favor boys 
at the elementary school level (Karcher, 2008). 

Further analyses in the BBBSA and SMP trials tested 
for potential differences in outcomes in association with 
characteristics of programs, mentor experiences, and 
mentoring relationships. These variables included infor- 
mation obtained after random assignment of youth to 
study condition. Efforts to determine whether programs 
worked more or less well as a function of such variables 
are non-experimental and subject to potential bias (Shad- 
ish, Cook, & Campbell, 2002). Consider, for example, 
that in the BBBSA and SMP trials mentor reports obtained 
at the end of the school year were utilized to assess the 
levels of training and support that mentors had received. 
Such reports could have been shaped by the impressions 
that mentors had of the degree to which their mentees 
were benefitting from the program, thus creating the 
potential for erroneous conclusions when using the data 
to attempt to gauge the impact of program training/sup- 



* These analyses were limited to programs that exclusively assigned all youth to 
either receive adult or high school mentors, thus addressing potential sources 
of bias associated with programs having the option of assigning youth to one 
or other type of mentor following random assignment (Herrera, Kauh, Cooney, 
Grossman, & McMaken, 2008). 



port. With this important caveat in mind, it is nonetheless 
noteworthy that the BBBSA trial found evidence of more 
favorable effects when adults rather than high school 
students were used as mentors at participating schools.* 
Outcomes favoring the treatment group were also more 
apparent when mentors reported receiving greater levels 
of support and greater access to school resources and 
space (Herrera et al., 2007). 

In the first year of the BBBSA trial, youth with the 
longest matches showed the greatest number of favorable 
outcomes as gauged by change from baseline when com- 
pared to control group youth (Herrera et al., 2007). Simi- 
larly, youth who reported feeling relatively closer to their 
mentors exhibited greater improvements in teacher-rated 
class work quality and youth-reported truancy (Herrera et 
al., 2007). In the SMP trial, however, site-level indices of 
mentoring relationship duration (% lasting six months or 
longer) and intensity (total hours of meetings per month) 
exhibited associations in mixed directions with outcomes 
(Bernstein et al., 2009). As noted, these types of findings 
must be interpreted cautiously because of the potential 
for variables of interest to be confounded with other ex- 
planatory factors, such as pre-existing attributes of youth 
(Grossman & Rhodes, 2002). 
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Meta-analytic Synthesis of Findings 

We now turn to our integration of findings across the studies 
using meta-analysis (Cooper, 2010; See Wheeler, DuBois, St 
Keller, 2010 for a detailed methodological description and 
complete findings). Combining findings across studies using 
meta-analysis can yield more reliable and precise estimates 
of program impact than is possible for any individual study 
examined in isolation (Lipsey & Wilson, 2001). Thus, “true” 
areas of program impact and those of more questionable 
validity may be brought into greater relief. In a typical 
meta-analysis, the findings 
of all applicable studies 
are coded in terms of a 
common metric referred to 
as an “effect size.” Here, 
we use Cohen’s d, which 
describes the difference 
between two groups (for 
example, treatment and 
control) in terms of the 
standard deviation of the 
measure involved; illus- 
tratively, an effect size 
of .25 would represent a 
difference of one-quarter 
of a standard deviation 
on the outcome measure. 

The standardized effect 
sizes are then combined 
across studies. By restrict- 
ing our focus to findings 
from BBBSA, CIS-SA, and 
SMP studies, the present 
meta-analytic synthesis is in effect limited to large-scale (n 
> 500) randomized control trials of school-based mentoring 
programs. Due to the small number of studies, it is not pos- 
sible for us to exploit the additional capacity of meta-anal- 
ysis to test for differences in effect sizes as a function of 
methodological factors or program characteristics (Cooper, 
2010). Meta-analyses involving small numbers of studies can 
be useful, however, especially when the studies involved 
are similar in design and focus as is true of the three recent 
trials of SBM (for other examples of meta-analyses based on 
similarly small numbers of studies see Connolly et al. , 2000, 
and Reinecke, Ryan, & DuBois, 1998). 

To undertake our analysis, we first examined the 
specific outcomes measured in each study and identified 
19 instances in which the same or closely related con- 
struct was assessed in at least two of the three studies (a 



full list of these constructs is included in Figure 1). We 
then calculated the mean effect size for each outcome 
across the relevant studies. In doing so, we used effect 
sizes included in the individual study reports and, when 
necessary, those based on data obtained directly from 
the study authors. Our specific procedures for averaging 
effect sizes and evaluating their significance followed 
recommended practices as described by Lipsey and 
Wilson (2001) and the Cochrane Collaboration (Higgins 
& Green, 2008). As shown in Figure 1, findings revealed 

evidence of favorable 
overall program effects 
(p < .05, as indicated by 
95% confidence inter- 
vals that did not include 
zero) on six outcomes: 
truancy (.18), reported 
presence of a supportive 
non-familial adult rela- 
tionship (.12), perceived 
scholastic efficacy (i.e. , 
perceptions of one’s 
academic abilities; .10), 
school-related misconduct 
(.11), peer support (.07), 
and absenteeism (.07). 



Discussion 

Policy decisions regard- 
ing social and educa- 
tional interventions for 
youth, especially those 
made at the federal 
level, often begin-and sometimes end— with a funda- 
mental question: Is the approach effective? Related 
policy considerations include the following: Will it reach 
those in need? Is it affordable and sustainable? How can 
it be improved? What are the alternatives? Our review 
of the recent trials of school-based mentoring has ad- 
dressed only the first of these questions (i.e., effective- 
ness). In this concluding section, we discuss our findings 
in the context of the broader set of relevant policy and 
practice considerations. 



Making Sense of Available Findings 

The three trials represent a small but important sam- 
pling from the full range of school-based mentoring 
programs already in operation in communities across the 
nation. As illustrated by our examination of the par- 



Figure 1. 




Weighted Mean Effect Sizes and 95% Confidence Intervals 


Reduced Truancy - 






Non-Familial Adult Relationships - 






Perceived Scholastic Efficacy - 






Reduced School-related Misconduct - 






Peer Support - 






Reduced Absenteeism - 






Peer Self- Esteem /Self-Concept - 






Global Self Esteem - 






Future Orientation - 






Academic Self Esteem - 






Connectedness to School - 






General Misconduct - 






Parent Relationships - 






Science Performance - 






Social Studies Performance - 






Reading Performance - 






Teacher-Student Relationship Quality - 






Math Performance - 






Substance Use - 
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Weighted mean effect sizes and 95% confidence intervals (CIs) for the 19 


outcomes in the meta-analytic synthesis of findings from the BBBSA, CIS-SA, 


and SMP studies. 





ticlpating sites in the three studies, even those pro- 
grams sponsored by a particular organization, agency, 
or funding initiative vary across numerous noteworthy 
dimensions. Given such diversity, a simple answer as to 
whether school-based mentoring programs “work” will 
inevitably remain elusive. Indeed, reading the findings 
of each of the three studies with the respective inter- 
pretations provided by each set of authors, one could 
come to substantially differing conclusions about the 
effectiveness of this intervention strategy. 

However, our comparison across the studies of pro- 
gram models and their observed levels of implementa- 
tion, samples of agencies and participants, and research 
methodologies 
helps to clarify 
several fac- 
tors potentially 
contributing to 
the differences in 
reported findings 
and conclusions. 

Those of the SMP 
study call into 
question the ben- 
efits of school- 
based mentoring. 

Our analysis re- 
veals, however, that findings of this study may have been 
attenuated by a range of factors such as the absence of 
a well-delineated program model, a lack of additional 
implementation support for study agencies, and a rela- 
tively high percentage of youth in the intervention group 
who were never paired with a mentor. The findings of 
the CIS-SA study similarly may have been attenuated by a 
relatively low dose of mentoring received by the average 
youth and by variable effects across elementary and high 
school settings. Related considerations suggest poten- 
tial explanations as well for the more optimistic picture 
conveyed by the first year findings in the BBBSA trial 
study report. These include a reasonably well-specified 
program model, supplemental implementation support 
for study agencies, and greater attention to measur- 
ing proximal outcomes, such as teacher-rated classwork 
quality. Finally, an important factor that clearly contrib- 
uted to differing study conclusions was the application 
of varying criteria for designating findings as statistically 
significant. When any one study’s criterion is applied uni- 
formly across the studies, their results portray a notably 
more consistent message regarding program effects. 



Our effort to make sense of study findings, collec- 
tively, using meta-analysis was also informative. Favorable 
program effects were evident for six of the 1 9 outcomes 
examined. Effect sizes for these outcomes ranged from .07 
to .18 and thus are similar in magnitude to the average ef- 
fect size of .11 found for school-based mentoring programs 
previously in a meta-analysis based on several less method- 
ologically rigorous evaluations (DuBois et al. , 2002). Taken 
together, available findings thus suggest that one year of 
participation in a school-based mentoring program tends to 
have modest effects on selected youth outcomes. However, 
the findings for the second year of the BBBSA trial sug- 
gest that many of these effects may not persist over time 

without students’ 
continued partici- 
pation in mentor- 
ing programs or 
other supportive 
services. 

The policy 
goal of the fed- 
erally funded 
Student Mentoring 
Program was to 
support school- 
based mentoring 
programs that 
would improve academic outcomes and interpersonal 
relationships and reduce involvement in delinquency and 
gangs (Fernandes, 2009). Although grades or teacher-rat- 
ed performance in particular subjects was assessed in all 
of the studies and out-of-school delinquent or antisocial 
behavior was assessed in both the SMP and BBBSA trials, 
our meta-analytic synthesis of findings did not reveal 
evidence of favorable program effects in these areas. 
Program effects are apparent on outcomes that may 
protect against involvement in problem behavior or be 
antecedents for future academic success: truancy, absen- 
teeism, and school-related misbehavior. Program impacts 
in these areas may lessen the likelihood of school dropout 
and emergence of more serious problem behaviors (Garry, 
1996; Goldschmidt & Wang, 1999). Perceived scholas- 
tic efficacy, for which program effects also are evident, 
similarly has been linked to improvements in academic 
achievement (D’Amico & Cardaci, 2003; B. J. Zimmerman, 
Bandura, & Martinez-Pons, 1992), particularly among 
low-achieving students (Multon, Brown, & Lent, 1991). It 
seems reasonable that school-based mentoring programs 
as currently structured should have greater potential for 



Taken together, available findings thus suggest 
that one year of participation 
in a school-based mentoring program 
tends to have modest effects 
on selected youth outcomes. 
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yielding short-term effects on these types of outcomes in 
comparison to those such as academic achievement and 
delinquency. Impacts on the latter outcomes may tend to 
emerge only after more immediate gains in relevant areas 
of student attitudes and behavior are realized. According- 
ly, they may be most appropriate and realistic to frame as 
longer-term program goals. 

Our meta-analysis results also indicate that school- 
based mentoring helps to achieve the goal of improving 
the interpersonal relationships of participating students 
by fostering connections with supportive, non-familial 
adults and by enhancing support from peers. As noted 
at the outset of this report, resiliency research suggests 
the inherent value of introducing more positive adults 
into the lives of youth from at-risk backgrounds (Scales 
& Leffert, 1999). Indeed, whether or not school-based 
mentoring programs cultivate positive adult and peer 
relationships among participating youth may be critical 
to realizing their potential benefits. Consistent with this 
possibility, an analysis of data from the BBBSA trial found 
support for a model in which program participation af- 
fected school-related behaviors and outcomes primarily 
through its ability to directly and indirectly enhance the 
quality of the youth’s social relationships (Silverthorn, 
DuBois, Herrera, & Kauh, 2010). 



Comparing to Other School-based Interventions 

It is useful to consider the findings obtained to date for 
school-based mentoring programs alongside those obtained 
for other interventions that may be implemented in schools. 
The Experience Corps program evaluation referred to previ- 
ously reported impacts on two reading skills assessments 
(effect size = .13, .16; Morrow-Howell et al., 2009) that are 
in the upper range of the effects found for school-based 
mentoring (see Figure 1). In the Experience Corps program, 
volunteers receive specific training in tutoring students in 
literacy skills and meet with them several times a week 
throughout the school year. A recent meta-analysis of evalu- 
ations of volunteer tutoring programs in schools similarly 
yielded evidence of significant positive effects on academic 
achievement in the areas of reading (.30) and writing (.45; 
Ritter, Barnett, Denny, & Albin, 2009; see also Elbaum, 
Vaughn, Tejero Hughes, & Moody, 2000). 

Social and emotional learning (SEL) programs 
provide another point of comparison. Such programs are 
designed to help youth “acquire the knowledge, atti- 
tudes, and skills to: recognize and manage their emo- 
tions; set and achieve positive goals; demonstrate caring 
and concern for others; establish and maintain positive 



relationships; make responsible decisions; [and] handle 
interpersonal situations effectively” (Payton et al., 2008, 
pp. 5-6). The most common strategy employed in these 
programs is classroom-based programming, which usually 
takes the form of a specific curriculum or set of lessons 
delivered by the teacher (Payton et al., 2008). A meta- 
analytic review of 180 evaluations of universally-applied 
SEL programs in schools found evidence of significant pro- 
gram effects on youth outcomes in six domains: SEL skills 
(Cohen’s d = .60), attitudes toward self and others (.23), 
positive social behavior (.24), conduct problems (.23), 
emotional distress (.23) and academic performance (.28; 
Payton et al., 2008). These findings may be influenced by 
the fact that many of the evaluations were smaller-scale 
efficacy trials in which program developers or research- 
ers provided significant support or oversight to those 
charged with delivering the program. 

Overall, it appears that other types of interven- 
tions often implemented in schools can have somewhat 
stronger effects on academic and other outcomes than 
school-based mentoring programs as currently consti- 
tuted. This may be especially the case for programs that 
reflect greater intensity or academic focus and those 
that are delivered with a high level of fidelity. It is note- 
worthy, however, that the largest effect size observed 
in our synthesis of findings from the recent school-based 
mentoring trials was for the reduction of truancy (.18). 
Truancy reduction is also one of two positive outcomes 
that persisted through to the 15-month assessment in the 
BBBSA trial (Herrera et al., 2007). 

Relatively few interventions have been found effec- 
tive for keeping students engaged in school. At pres- 
ent, the U.S. Department of Education’s “What Works 
Clearinghouse” lists only six interventions as meeting or 
exceeding criteria for having potentially positive effects 
within the category of helping students stay in school 
(U.S. Department of Education, 2010). Mentoring strate- 
gies are a central component of three of these programs 
(Check & Connect, ALAS, and Twelve Together). A recent 
independent review of programs for preventing school 
dropout similarly highlighted mentoring as one of a lim- 
ited number of approaches associated with program ef- 
fectiveness (Tyler & Lofstrom, 2009). Interestingly, there 
is also evidence that youth may be more likely to attend 
school on the specific days when they will spend time 
with their mentors (Volkmann & Bye, 2006). It should be 
noted, however, that the programs identified above tend 
to make use of practices that are currently not typical in 
school-based mentoring programs. In the Check & Con- 
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nect program, for example, a paid para-professional serves 
as a mentor to several students and then makes daily con- 
tact with each of these students in the school setting (An- 
derson, Christenson, Sinclair, & Lehr, 2004). Nonetheless, 
in view of the encouraging signs that current school-based 
mentoring programs can foster school engagement, further 
strengthening of such programs could conceivably produce 
longer-term benefits in the area of school completion. 



Concluding Points 

Policymakers, practitioners, and others require high 
quality evidence as they weigh potential investments in 
school-based mentoring and other forms of intervention 
designed to benefit youth. Our analysis highlights several 
key areas of programmatic and methodological informa- 
tion that warrant careful consideration in the decision- 
making process. The merits of taking stock of the full 
landscape of available studies are also readily apparent 
from our review as are the corresponding hazards associ- 
ated with basing decisions on the findings or conclusions 
of a single investigation. Inevitably, too, as we have 
emphasized, evidence-based deliberations require simul- 
taneous attention to a host of other factors, including 
legislative and policy priorities, organizational resources 
and mission, and the relative cost-effectiveness of other 
available services and supports. 

For guiding the future development of school-based 
mentoring, the present analysis points to the following as 
priorities: 

• Longitudinal studies are needed that help ex- 
plicate the processes at work in school-based 
mentoring relationships as they develop over time 
and that provide information on how program 
practices can support this process. Such studies 
should investigate the role of youth and mentor 
characteristics, match longevity, and relationship 
quality in fostering positive outcomes for youth. 

They also should incorporate longer-term follow- 
up measurement so as to better gauge the impact 
of programs over time. 

• Innovative approaches to school-based mentoring 
should be investigated. These include, for ex- 
ample, a hybrid “school-plus” mentoring model in 
which mentors and youth have the opportunity to 
spend time together outside of the school setting 
(Harris, 2009), as well as programs including more 
structured components. Evaluation of program 
models also should examine the role of program 
fidelity as a moderator of effectiveness. 



• Cost-benefit analyses should be integrated into 
future program evaluations. Such analyses are 
needed to clarify the extent to which observed im- 
pacts offer benefits whose value exceeds program 
expenditures. They are also necessary to advance 
understanding of the cost-effectiveness of school- 
based mentoring as compared to alternative forms 
of intervention. 

For policymakers and practitioners considering 
future investments in school-based mentoring programs, 
our review offers the following for consideration: 

• School-based mentoring programs as currently 
constituted appear to have significant, but rela- 
tively small, effects on several outcomes related 
to school success. It is not clear that these ef- 
fects extend to indicators of academic perfor- 
mance per se, such as grades, but rather are 
concentrated around more proximal behaviors 
and beliefs that keep students engaged in school 
and that are likely to foster learning. 

• Effect sizes observed for SBM are in a range that 
makes their interpretation subject to underly- 
ing perspectives and priorities. Policy-makers 
considering various options for investment in 
school-based prevention have reason to be 
somewhat skeptical and to call for more con- 
vincing evidence, whereas practitioners de- 
veloping and delivering this intervention have 
reason to claim that SBM can achieve positive 
results and to be optimistic about the possibility 
of further improvements. 

• Given the diversity of school-based mentoring 
models and programs and the fact that mentor- 
ing is an individualized intervention, planners 
must consider which model will work for which 
students under which circumstances. However, 
the existing evidence addresses the effective- 
ness of SBM at only a very general, aggregated 
level. Meta-analytic results provide initial sup- 
port for the concept of school-based mentor- 
ing as defined by basic features that cut across 
many programs. Nevertheless, specific programs 
are likely to vary substantially in their effective- 
ness according to their design, target popula- 
tion, and quality of implementation. This reality 
underscores that the results of the three recent 
randomized control trials of school-based men- 
toring cannot substitute for continuous monitor- 
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ing and evaluation of outcomes within specific 
programmatic contexts. 

In the past decade, school-based mentoring has 
undergone rapid expansion and has attracted significant 
attention from researchers. Few interventions devel- 
oped so recently have amassed evidence from three 
large and rigorous randomized trials. Our integrative 
review has drawn on these trials to examine the cur- 
rent state of knowledge regarding the effectiveness of 
school-based mentoring as an intervention that seeks 
to enhance resilience among youth from at-risk back- 
grounds. As further research emerges, policy-makers 
and practitioners will undoubtedly learn more about the 
potential of this intervention to promote youth develop- 
ment and academic success. ■ 
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Commentary 

After-School ime 

Jodie Roth 

Columbia University 



A merican school children 
spend a larger propor- 
tion of their waking 
hours in discretion- 
ary activities than in 
school. Students have 
a range of choices for how to fill these 
hours, including watching TV, hang- 
ing out with friends, or engaging in an 
organized sports or arts activity. These 
choices have developmental conse- 
quences, some positive and some nega- 
tive. The choices available to parents 
and students for the non-school hours 
are shaped, in part, by the decisions of 
policymakers and funders, who choose 
to fund some programs or activities 
over others. This issue of the Social 
Policy Report provides valuable infor- 
mation about one of those choices: 
school-based mentoring. 1 The purpose 
of this commentary is to acquaint the 
reader with findings from the broader 
landscape of organized out-of-school 
time choices. 

The belief that participation in 
organized activities during the non- 
school hours is beneficial to youth is 
widespread, although research evi- 
dence is mixed. Organized activities 
refer to any activity or program for a 
group of youth that is supervised by 
an adult and meets on a regular basis 
outside of school hours. Organized 
activities include things such as school- 
based extracurricular activities and 
after-school programs. Although some 



'Some of the mentoring activities in the school-based 
mentoring programs reviewed by Wheeler, Karcher, 
and DuBois occurred after school and some occurred 
during the school day. 



organized activities explicitly include 
mentoring activities similar to those 
in school-based mentoring programs, 
in most it is a more informal part of 
the activity. Our research has dem- 
onstrated that adolescents generally 
benefit from more participation in or- 
ganized activities (e.g., Gardner, Roth, 
£t Brooks-Gunn, 2008). For the vast 
majority of teens, greater amount and 
variety of participation confers added 
benefits. The findings also suggest that 
greater participation in organized ac- 
tivities has positive effects on youths’ 
experiences in school and with peers, 
which in turn contributes to better 
developmental outcomes. Research on 
organized activities suggests that pro- 
cess features, which include personal 
relationships between participants 
and the staff as well as among partici- 
pants, are critical to developmental 
growth among participants (e.g., Roth 
£t Brooks-Gunn, 2003). 

One type of organized activ- 
ity, after-school programs, receives a 
great deal of scrutiny with regard to 
outcomes because of the recent pro- 
liferation in publicly funded programs. 
After-school programs typically include 
both academic assistance and various 
enrichment activities. As with school- 
based mentoring research, findings are 
mixed across studies, with some high 
profile null results contributing to cuts 
in funding. A recent meta-analysis of 
75 evaluations, however, shows the 
positive academic, behavioral, and 
social benefits from participation in 
an after-school program. It also allows 
for comparison with the effect sizes 



presented in this report. It is important 
to note that the only studies included 
were those of after-school programs 
where one of program goals was the 
development of one or more personal 
or social skills such as problem solv- 
ing, conflict resolution, or leadership. 
Significant mean effects were found for 
achievement test scores (.17), school 
grades (.12), self- perceptions (i.e., 
self-esteem; .34), school bonding (.14), 
positive social behaviors (.19), and 
fewer problem behaviors (.19; Durlak, 
Weissberg, & Pachan, 2010). The ef- 
fect sizes were larger among programs 
utilizing a sequenced, focused, and 
explicit approach to teaching skills that 
included active learning. 

School-based mentoring offers one 
avenue for addressing the developmen- 
tal needs of students. Yet as this brief 
review shows, other activities also help 
promote youths’ positive development, 
including their academic achievement. 
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E vidence on the eco- 
nomics of mentoring 
is limited, but it does 
demonstrate that 
mentoring programs are 
relatively inexpensive. 
This point is important; the program 
would not need enormous effects 
to demonstrate cost-effectiveness. 
Fountain and Embreton (2000) esti- 
mate program costs and generally 
conform to the practices of eco- 
nomic evaluation-that is, the study 
seeks to calculate the marginal (or 
additional) cost for offering mentor- 
ing to a youth. The authors report 
calculations from a sample of 50 
mentoring programs selected from a 
database of 720 programs that had 
participated in an earlier study. The 
average mentoring program costs an 
estimated $1,114 per participant, 
but those costs varied enormously 
across programs. The Wheeler, 

Keller, and DuBois paper demon- 
strates that future economic analy- 
ses would have to consider variation 
in both effectiveness and costs. 

An open question is whether more 
costly programs are more effective. 

How strong is the evidence on 
the cost-effectiveness of mentoring 
programs? A fair assessment would 
be that existing studies should be 
considered only suggestive. They 
indicate the possibility that mentor- 
ing is cost-effective but offer no 
strong evidence because they omit 



key aspects of standard economic 
analyses. 

Existing economic evalua- 
tions include Belfield (2003). This 
study combines these cost estimates 
with data from the impact study of 
the Big Brothers Big Sisters (BBBS) 
community-based mentoring pro- 
gram (Grossman & Tierney, 1998). 
This study was similar in design 
(e.g., wait-list controls) and mea- 
surement to the evaluation of the 
BBBS school-based mentoring pro- 
gram described by Wheeler, Keller, 
and DuBois. The overall results were 
very similar as well. Although the 
Belfield study has weaknesses (for 
example, it seems to muddle the 
taxpayer and social perspectives), 
it also has many strengths. (For a 
fuller discussion of the economics 
of mentoring, see Foster, in press.) 
These include an effort to capture 
a fuller range of the broader, public 
benefits of education, including the 
broader social returns (Haveman 
& Wolfe, 1984). Largely the study 
attempts to link a series of observed 
outcomes to unobserved future 
outcomes to which dollar values 
can be attached. For example, the 
author uses data from the National 
Longitudinal Survey of Youth to link 
truancy to long-term earnings. The 
author then uses that relationship 
to express the impact of the BBBS 
program on truancy in dollar terms 
(i.e. , future earnings). 



Combining data in this way is a 
frequent practice in economic evalu- 
ation. Obtaining the information 
needed for an economic evaluation 
from a single study or evaluation is 
difficult. The primary problem with 
this practice, however, is that each 
of these projections introduces po- 
tential errors of one sort or another. 
For example, even using a large 
dataset, projecting future earn- 
ings from education is an uncertain 
endeavor. That uncertainty intro- 
duces further uncertainty into the 
calculated economic return. Belf- 
ield neglects this issue entirely. In 
sum, this study is suggestive of the 
potential for mentoring and cost- 
effectiveness, but at this point the 
findings are only suggestive. They 
are not a valid basis for allocating 
public funds. 

Two additional studies pro- 
vide calculations involving programs 
and dollars but do not meet the 
minimum standards for economic 
evaluaton. Both represent “back-of- 
the-envelope” calculations. Blech- 
man and colleagues (2000) calculate 
cost-effectiveness ratios for two 
programs offering mentoring ser- 
vices. However, they use the amount 
of a state block grant for program 
costs; as discussed above, this likely 
underestimates true program costs. 
No confidence intervals are provided 
for the cost-effectiveness ratios, and 
the study suffers from other flaws. 



Social Policy Report V24 #3 



23 



School -Based Mentoring 




A second study by Aos and col- 
leagues (2004) combines data on the 
estimated costs and benefits of sev- 
eral different mentoring programs. 
These authors and colleagues (Aos, 
Lieb, Mayfield, Miller, & Pennucci, 
2004) follow much the same meth- 
odologies as those described above, 
but do draw on more data sources. 
They also report not just the benefit- 
cost ratio but also the net present 
value. All in all, however, the study 
has many of the problems of earlier 
studies. For example, the study lacks 
essential elements of an economic 
analysis, such as any sense of the 
uncertainty with which the costs and 
benefits are estimated. 

In sum, existing research 
demonstrates that the costs of 
mentoring programs are variable and 
rather modest, at least in compari- 
son to the high costs of delinquency, 
school dropout and substance abuse 
(Cohen, 1998). Simple comparisons 
reveal that the latter dwarf the 
former. Nonetheless, at this point, 
the literature does not offer the sorts 
of comprehensive economic analysis 
that are required to judge mentor- 
ing cost-effective and to suggest that 
public funds should be expended on 
such programs. 
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heeler, Keller, 
and DuBois 
suggest that 
policymakers 
have reason 
to be skepti- 
cal and practitioners have reason to 
be optimistic about the impact of 
school-based mentoring. Contrarily, 
the findings leave this practitioner in 
the skeptical camp. 

The authors point out that 
school-based mentoring models 
have been attractive to agencies for 
reasons such as ease of volunteer 
recruitment and lower costs. School- 
based mentoring also allows programs 
to more easily target specific student 
populations and limit liability issues. 
Unfortunately, this rationale seems to 
make things easier on the adults run- 
ning programs and may compromise 
the impact of mentoring on the youth 
served . The data reviewed here does 
not suggest to me that the benefits to 
students are significant. 

There are positive, if not 
surprising, results that are related 
to having a school-based mentor. 

The reduction in truancy is clearly 
the high point of the report, and 
that alone may make school-based 
mentoring a valuable strategy. Along 
with the truancy finding, the positive 
impact on students’ feelings of adult 
relationships, scholastic efficacy, 
and school-related misconduct are 
logical outcomes of having another 
role model who is checking in with a 
student at school. 



School administrators may 
be skeptical because the review of 
studies found no impact on any area 
of academic performance. This is 
not surprising because the models 
studied did not emphasize academic 
skill building or have any direct 
connection to classroom instruc- 
tion. Even when positive impacts 
were found, the effect sizes were 
relatively modest when compared 
to social and emotional learning 
programs and volunteer tutoring 
programs. In an era of heightened 
accountability, school leaders may 
not be interested in an intervention 
that does not produce improve- 
ments in student achievement. 

Mentoring programs and 
schools that still want to implement 
models similar to those analyzed 
here may pick up some important 
guidance from the authors’ excellent 
comparisons to existing best practice 
recommendations and research. As 
the authors note, the duration of the 
match between mentor and mentee 
is important to effective mentor- 
ing relationships. The fact that the 
average duration of matches in these 
studies was less than six months rais- 
es concern. There is some evidence 
from evaluation of community-based 
mentoring programs that matches 
lasting less than six months may do 
more harm than good, especially 
if students expected the match to 
continue longer (Rhodes & DuBois, 
2006). School-based mentoring pro- 



gram designers may want to follow 
best practice guidelines that suggest 
asking mentors to make a minimum 
one-year commitment (MENTOR, 
2009) and even consider support- 
ing matches beyond the one-year 
benchmark. 

Finally, the authors cite 
the need for more research and 
acknowledge that they have not 
addressed a series of deeper ques- 
tions about school-based mentoring 
models. Prior to the next generation 
of research, policymakers, funders, 
and mentoring agencies should 
recognize that school-based mentor- 
ing as configured in these studies 
may not be structured in a way to 
create strong effects. If the field 
evolves and school-based mentor- 
ing programs meet higher standards 
for quality, then perhaps we will see 
stronger effects over time. But I am 
skeptical that as long as programs 
are designed for ease of implementa- 
tion, they will be adult-focused and 
will fail to have the intended impact 
on students. 
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